Using Clinical Vignettes to Evaluate VTE Protocol Adherence

BACKGROUND Venous thromboembolism (VTE) prophylaxis is underutilized in hospitalized medical patients. Underutilization might occur as a result of resident practice variation incurred by using a complex risk assessment tool. OBJECTIVE To examine what impact repetitive exposure to an electronic point-based VTE risk assessment tool has on resident inter-rater reliability and protocol adherence. DESIGN Pre and post intervention cross-sectional cohort study. SETTING Single academic center. PATIENTS Convenience samples of Internal Medicine residents. INTERVENTIONS Residents completed clinical vignettes before and after any exposure to an electronic risk assessment tool and reminder alert. They were asked to make three determinations using a point-based VTE risk assessment tool: risk stratification, identify contraindications, and VTE prevention strategy. MEASUREMENTS Inter-rater reliability for risk assessment, contraindications, and VTE prophylaxis strategy and protocol adherence. RESULTS Kappa scores for VTE risk assessment did not change, but improved for VTE plan increasing from 0.28 to 0.37. Protocol adherence improved from 71% in 2008 to 79% (P = 0.06). There was a significant decrease in under-prophylaxis (22% to 6%, P < 0.0001) but a significant increase in over-prophylaxis (7% to 16%, P = 0.001). CONCLUSIONS Using clinical vignettes, we determined that daily exposure to an electronic risk assessment tool did not improve the inter-rater reliability of a point-based risk assessment tool when used by medical residents. This might be due to inexperienced providers using a complex point-based tool. Overall, adherence improved, and under-prophylaxis decreased, but over-prophylaxis increased. Clinical vignettes are a generalizable method to monitor resident prophylaxis practices and way to identify educational and process improvement opportunities. KEYWORDS Resident; Inter-rater reliability; Venous thromboembolism; Agreement; Risk assessment score.